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(57) A note-taking system based on a notepad 
computer with an integrated audio/video-recor- 
der is described. A document is created or 
retrieved. As the user types on the keyboard or 
writes with the stylus or similar input instru- 
ment, each character or stroke that is input by 
the user is invisibly time-stamped by the com- 
puter. The audio/video stream is also continu- 
ously time-stamped during recording. To play a 
section of recording back, the user selects part 
of the note (perhaps by circling it with a stylus) 
and invokes a "playback selection" command. 
The computer then examines the time-stamp 
and "winds" the record to the corresponding 
place in the audio/video recording, where it 
starts playing - so that the user hears and/or 
sees what was being recorded at the instant the 
selected text or strokes were input. With a 
graphical user interface, the user may input key 
"topic" words and subsequently place check 
marks by the appropriate word as the convers- 
ation topic veers into that neighborhood. 
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This invention relates to a data access system, 
and in particular to one in which the taking of notes 
usually accompanies the recording of data. The notes 
themselves are used to gain selective access to the 
data. 

The problem 

At seminars, interviews or meetings, it has long 
been recognised that the simple act of taking notes 
helps the writer memorize key facts. On the other 
hand, to make a note, the listener has to divert atten- 
tion from the speaker, figure out how to encode the 
information he has heard or the idea he has had, and 
then focus on writing it down. It doesn't seem surpris- 
ing that the listener often loses the speaker's thread! 
Nowadays many people resort to making audio or 
video recordings of important events and then spend- 
ing time transcribing key Ideas afterwards. Locating 
the interesting parts of an audio or video recording is 
often very time-consuming and tedious, which 
reduces the likelihood that the listener will bother to 
transcribe the tape, or even bother to make the 
recording in the first place. 

In many situations the need to record is not anti- 
cipated. Only after the event, sometimes long after- 
wards, does the need become evident To overcome 
this problem it may eventually become technologi- 
cally possible, from a storage point of view, to record 
and store every second of a person's working day. 
The problem of locating key pieces of information from 
this huge and expanding base of recordings becomes 
overwhelming. 

This invention builds upon these ideas by provid- 
ing a semi-automatic, fine-grained, audio/video index- 
ing tool. Although the invention focuses on gaining 
access to audio and video records, it is not limited to 
this application. It is envisaged that the invention 
could also be applied to accessing files in a computer 
system, phone call records, or any other time-stam- 
ped data set 

The present invention will now be described by 
way of example with reference to the accompanying 
drawings, in which: 

Figure 1 is a diagrammatic view of one form of the 

invention using a laptop computer; 

Figure 2 is a diagrammatic view of another form 

of the invention using a notepad computer with 

stylus input, and 

Figure 3 is a diagrammatic view of one form of the 

architecture of the invention. 

The example of the invention shown in Figure 1 
combines a laptop computer 2 with a video or audio 
recording device 4. 

In the example of the invention shown in Figure 
2, the portable computer is replaced by a note-pad 
computer 6 with stylus or other graphical input device 
8, connected to video or audio recording device 4. The 



stylus input device may take the place of the keyboard 
of the portable computer as the user's means of input. 
Alternatively, a keyboard 10 may be connected to the 
notepad computer for input 

5 Either form of computer is interfaced to the recor- 

der in such a manner that the computer can monitor 
the recorder's progress during recording. To enable 
this, the recorder may emit time-code or make other 
unique index information continuously available to the 

10 computer. Additionally the computer can operate the 
recorder directly, causing it to record, or play from a 
specific index point as required. In any situation it is 
possible for the computer to find out the time-code or 
index information. 

15 In the general system shown in Figure 3, the com- 
puter presents a document editor style user interface 
to the user. This Interface may resemble that of a 
word-processor, an illustrator, or an application speci- 
fic interface, depending on the user's needs and com- 

20 puter hardware. The user interface provides 
commands for creating a new document and starting 
and stopping recording. 

A new document is created, or an old one recal- 
led, and recording commences. The editor 22 allows 

25 the user to draw, type or sketch on the blank docu- 
ment using keyboard 10, stylus 8, or other graphical 
input device. As each mark (or indicium) 26, is added 
to the document (an 'indicium' is any indivisible dis- 
ptayable symbol created by the interface, e.g. pen- 

30 stroke, character, graphical symbol, etc.), the editor 
time-stamps and stores it in an indicium-to-timestamp 
index 14. The video-frame time-stamper 20 also con- 
tinuously notes the index information 32 arriving from 
the recorder 4 at that instant, and time-stamps it and 

35 stores it in a timestamp-to-timecode index 16. The ti- 
me-stamps are not visible to the user - they are stored 
with the computer's internal representation of the indi- 
cia. When the session is over, the user stores away 
the document; the contents of the two indexes, and 

40 the recording for subsequent use. Any method that 
allows time-stamps to be used as an index into the 
recording may be used instead of the timestamp-to-ti- 
mecode index, e.g. interpolation of the timecode itself. 
When the user wants to recall sections of the 

45 recording, the appropriate related document is ret- 
rieved and recalled to the display 30 by the browser 
24. Using the stylus 8, or the keyboard 10, the user 
selects one or more indicia on the document, perhaps 
by circling them, positioning a cursor, or typing an 

so identifying name, and instructs the browser 24 to play 
the associated video section. The browser 24 iden- 
tifies the selected indicium or indicia, looks up the ti- 
me-stamp^) in the indicium-to-timestamp index 14; 
looks up the timestamp(s) in the timestamp-to-timeco- 

55 de index 16, and plays the section of the recording in 
the area indicated by the resultant timecode. Thus the 
user sees what was recorded at the time the marks 
were made on the video monitor 28. 
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Indicium-to-timestamp and timestamp-to-timeco- 
de indices may be combined into a single indicium-to- 
timecode index. 

It is expected that users will develop shorthand 
notations for marking the document to indicate an s 
interesting idea, change of topic or speaker, and so 
forth. It also is expected that users will type or hand- 
write key words as the topic arises, and then add 
additional marks nearby each time the subject veers 
in the direction of that topic. Thus, to hear all section 10 
of a seminar associated with the same topic, it is 
necessary only to circle all the marks in the location 
of the topic keyword. 

Accordingly it will be seen that the present inven- 
tion provides a simple, Inexpensive, lightweight, low- is 
power device for quick access to data records. 



Claims 

20 

1 . A system for providing random access to a time- 
stamp identified part of a time-stamped data set, 
said system comprising: 

means for capturing, filing, retrieving, dis- 
playing, and automatically time-stamping seer- 25 
generated indicia for selection by a user, said 
time-stamped data set and said time-stamped 
indicia having a common time base; 

means for determining the time-stamp of 
each user selected indicia; and 30 

means for indexing into said data set to 
identify the part thereof that is time-stamp corre- 
lated with said selected indicia. 

2. A system as claimed in claim 1 , including means 35 
for presenting to the user the identified portion of 

the data set. 

3. A system as claimed in claim 2 f in which the time- 
stamped data set is audio and/or video data. 40 

4. A system as claimed in any of claims 1 - 3 , which 
is implemented by means of a portable computer. 

5. A system as claimed in any of claims 1-3, which 45 
is implemented by means of a portable video 
recorder. 

6. A system as claimed in claim 5, in which the port- 
able video recorder is a camcorder. so 

7. A system as claimed in claim 4 in which the port- 
able computer is a notepad computer with stylus 
input 
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